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Abstract 

An important method of evaluating students’ progress in courses is the administration 
of tests. The increasing availability of handheld computers, such as Palm and Windows 
CE devices, in conjunction with wireless networks, allows the automating of aspects 
of giving tests in the classroom. During the Spring 2000 academic semester, we exper¬ 
imented with using Windows CE devices in a chemistry course to allow the instructor 
to intersperse, with lecturing, the administration of a form of “concept tests”, in order 
to determine whether material just covered was understood, thereby enabling the in¬ 
structor to modify the content or presentation of the rest of the lecture. We found that 
most students preferred the use of handhelds for this purpose to the use of a show of 
hands or holding up of flashcards. 
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1 Introduction 


An important method of evaluating students’ progress in courses is the administra¬ 
tion of tests. Tests come in many different forms, from simple true-false tests, and 
multiple-choice and fill-in-the-blank tests, to more free-form tests involving essays. 
The simplest forms of tests are easy to grade mechanically, so as technology becomes 
more and more used in education, it seems reasonable to use it to aid in the creation of 
tests, and grading and analyzing students’ results. 

The most obvious way to immediately make use of technology is to simply treat 
it as an improved medium for doing what was done before. In this scenario, let’s call 
it “ordinary” testing, a student logs into a course’s test Web site and is presented with 
a simple test, e.g., multiple-choice, and after the student submits a completed form, 
the test is graded, statistics are gathered, and feedback is given to the student. This 
scenario amounts to a more efficient method of giving the old paper-and-pencil test. 

It is interesting to consider, however, ways of using technology to do new things, 
rather than just do old things better. With the increasing availability and use of handheld 
computers, such as Palm and Windows CE (now Pocket PC) devices, and wireless 
networks, it has now become possible to take advantage of in-class use of computers 
and immediate feedback to the instructor and students. So here is another scenario for 
testing: a instructor with a server PC creates some kind of test, e.g., multiple-choice 
or fill-in-the-blank, and loads it into a program that will cause students with PDAs to 
receive a representation of the test and submit responses. The server then collects all 
the data, tabulates the results, computes statistics, and displays them to the instructor, 
who then makes use of the information to continue the lecture. The main point of 
this accelerated form of pop quiz, a version of what is known as a “concept test” [1], 
is to enable the instructor to determine whether material just covered in lecture was 
understood; so, for example, if it turns out that it was not understood, the instructor 
could elaborate on the material and slow down the lecture as appropriate. 

In summary, the work dealt with two kinds of tests: 

Ordinary test A test designed to work as a one-time quiz that can be taken and scored 
just once. 

Concept test A test designed to be taken as part of a lecture, with immediate feed¬ 
back displayed to the students and instructor, in which a student may repeatedly 
submit new responses to the same question. 


2 Approach 

We decided that instead of using the Pebbles 1 PDA/PC technology, we would use off- 
the-shelf Web technology. The main advantage of doing so is a kind of portability . 
Instead of supporting just Windows CE, our first intended platform, we can support 
any combination of platforms that involves running a Web server and running Web 
clients. Example clients other than handheld Windows CE devices would be desktop 
PCs, laptops, and in the future, Palm devices. 

*http: //www. cs . emu. edu/"'pebbles/ 
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We originally began by developing software to aid in the creation of custom on¬ 
line “ordinary” tests, with arbitrary questions, answers, and accompanying content, 
then worked on modifying the software in light of collaborating with an undergraduate 
chemistry course to demonstrate possible innovative uses of handhelds in the classroom 
given the infrastructure set in place by the Wireless Andrew project 2 . 

The course Chemistry 09-106: Modem Chemistry IP was taught by David Yaron 
and Garry Wamock. It turned out that Prof. David Yaron, of the CMU chemistry de¬ 
partment, for the purposes of his course, only needed and wanted specific kinds of 
“concept” tests, so we specialized the just-developed concept test software to fulfill his 
needs. This specialized concept test, call it the “generic concept test”, is a variant of the 
concept test in which there is only one question, and no question/answer uploading by 
the instructor is necessary, because questions and choices are given in lecture through 
paper handouts distributed before class. The instructor can reset for the next question 
immediately. 

3 Hardware 

The handheld we used was the HP Jornada 680, since HP had donated many units. But 
actually, the software is platform-independent, since it uses standard Web CGI. The 
only platform-dependent issues that came up were the following: 

• strange behavior in Microsoft Pocket Internet Explorer having to do with choice 
CGI field name (this did not occur with the older handheld HP 620LX, used 
before the new ones arrived), and 

• the desire to lay out the GUI elements in such a way as to comfortably fit in the 
limited screen size of the Jornada handheld. 

The server software was all developed and run on a Linux PC, running CMU facil- 
itized Red Hat 4.2. 


4 Administration 

The distribution of handhelds to the students of the chemistry course were handled by 
the CMU Computer Store. 

Unfortunately, a mapping of handheld unique IP address to student was not avail¬ 
able, so we had to have students “register” their handhelds in class. We did this by 
providing a Web page in which students entered their Andrew IDs through their hand¬ 
helds, and building our own table of which student had which handheld. 

A Linux PC, dahmer .pscico. cs . emu. edu, used as the Web server, was 
aliased to peb. cs . emu. edu, to make it easier for students to remember and ac¬ 
cess. The Web server on this machine is Apache 4 , which comes installed and ready to 
go on Red Hat Linux. 

2 http://www.cmu.edu/computing/wireless/ 

3 http://ir.chem.cmu.edu/cheml06/ 

4 http;//www.apache.org/ 
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5 Classroom use 


There were 98 students present in each lecture, in the earlier stages of the course. 

Each day on which concept tests were given in the class, approximately six tests 
were presented to the students. 

The number of students who responded during the concept tests varied: sometimes 
there were as few as 20, sometimes as many as 50, depending on the day. Many 
students either did not have their handhelds with them; did not have them out of their 
backpacks; or did have them, but chose not to participate in the concept testing. 

During the first trial uses of the handhelds for concept testing, a few students played 
around and deliberately changed answers repeatedly, or chose “None”. Eventually this 
behavior disappeared. 

Eventually, in order to collect statistics for each student, we had the students “reg¬ 
ister” their handhelds, so that it could be determined who was taking concept tests, and 
what their answer patterns were (Figure 1). It might have been better if registration 
could have been handled as the handhelds were being assigned to students. 

It was interesting to see how Prof. Yaron worked the concept tests into the lecture. 
He would pose and explain a question, and students would start answering. After 
perusing the results for a while, he would then start to explain the answer, or remark 
that, for example, most of the people were settling on an answer that was incorrect, 
with the result that the distribution of newly obtained student responses would change. 

During two of the lectures, Prof. Yaron saved out score data for a completed test 
before moving on to the next test. The total number of tests logged during these two 
lectures was 11. There were several days of tests from previous lectures that were not 
logged, because the logging feature had not yet been implemented or had not yet been 
tried. 


5.1 Concept tests in conjunction with a demonstration 

Several times concept tests were given interleaved with a chemistry demonstration. 
The chemistry demonstration worked as follows: Prof. Wamock came prepared with 
test tubes, chemicals, torches, etc., and when Prof. Yaron turned over control of the 
classroom to Prof. Wamock, the projector was switched to take input from a camera 
directed at the demonstration area rather than from Prof. Yaron’s laptop. 

This seemed to work well. The demonstrations were enjoyable, and involved highly 
visible consequences of chemical reactions, e.g., soda bottle rockets being launched, 
light bulbs turning on, gases forming, liquids changing color. Prof. Wamock would 
pause his demonstration at strategic points, e.g., preparing substances but not yet mix¬ 
ing them, whereupon control would pass to Prof. Yaron, who would give a concept test 
asking the students to predict the consequence of the next step. After enough submis¬ 
sions, he would then pass control over to Prof. Wamock, who would then continue the 
demonstration. 
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Hi Netscape: Register Handheld 

File Edit View Go Communicator 
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Your IP address is 128.2198.228. 

Andrew ID (without + or fcandrev. emu edu): 


Figure 1: Page for students to register their handhelds 
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6 Software 


There are two classes of users: 

• The student, who sees a test pop up and submits responses. 

• The instructor, who creates a test and loads it into the system and initiates a 
testing session. 

Correspondingly, there are two separate ways of accessing the test data: 

• http://peb.cs.cmu.edu/ (Figure 2) is for students. From this page stu¬ 
dents may 

- take the generic concept test; 

- take a particular concept test; 

- take a particular ordinary test. 

• http: //peb. cs . emu. edu/admin/ (Figure 3) is for the instructor. The 
instructor may 

- list registered handhelds; 

- monitor the generic concept test; 

- display generic concept test scores; 

- upload a test; 

- monitor a concept test; 

- monitor a ordinary test. 

For both the student and the instructor, the user interface consisted of Web forms 
with buttons, text fields, radio buttons, tables, etc. 

We were not sure whether the Web server and CGI programs would perform suffi¬ 
ciently efficiently, or whether 80 handhelds in a room would overwhelm the network, 
but there seemed to be no problems. 

The organization of the course information is that each course has its own directory, 
and each test has its own subdirectory under a course. Another organization (some kind 
of graph rather than a tree) would be needed if one wished to share a test or related data 
among multiple courses, for example. 

Some access control was provided to prevent students from using the Web site 
intended for instructors: Apache’s htaccess feature was used so that only those 
knowing the correct login and password, e.g., the instructor, would be able to access 
the Web page. 
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Figure 2: Main Web page for students 
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Figure 4: Login page for an ordinary test 
























Figure 5: Login page for a concept test 
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6.1 Take test 


The main Web page for students is a login form. There are two different pages, one for 
ordinary tests (Figure 4) and one for concept tests (Figure 5). The generic concept test 
is just a particular kind of concept test that bypasses the login form (Figure 6). 

The student, upon logging into a particular course and test, sees formatted content 
that is an interleaving of the HTML that the instructor designed and uploaded, and a 
form where the question/answer content can be specified. For example, in Figure 7, the 
student has successfully logged into a concept test. 

The form consists of a numbered list of items, each of which has question text 
(which has been specified by the instructor with HTML) and a lettered list of possible 
answers (also specified with HTML), with a radio button preceding each answer. There 
is a “submit” button at the bottom of the page, and for usability reasons, a duplicate 
submit button at the top of the page (to make it immediately visible on the handhelds, 
which have very limited vertical space). There is also a “reload” button at the top of 
the page, which must be pressed when a new test begins, because it may have different 
content from the old one; an attempt to submit a form for an old test fails because each 
test is tagged by an ID that is kept track of. 

If the student does not answer a question, this fact is detected, and shows up as the 
response “None”. 

After the student has submitted responses for the current test, the result differs de¬ 
pending on whether the test is a concept test or an ordinary test. If it is a concept test, 
the new responses are logged in the test score database, without overriding old re¬ 
sponses, and a page appears with the same concept test being displayed, except this 
time there is a note indicating the last selected response, in case the student wishes to 
choose a different one before the instructor stops the test (Figure 8). On the other hand, 
for an ordinary test (Figure 9), the submission process halts, and a page appears that 
indicates for each question the student’s response and the correct response (Figure 10). 

6.2 Submit test 

XML was used as a repository of test information, to avoid the need to needlessly 
invent special syntax for learning and parsing. We designed an XML DTD test. dtd 
to specify the syntax of the annotations. 

However, it was not practical for complete online content to be written in XML, 
e.g., an online test will typically have an introduction, images, and interspersed for¬ 
matted content, which would be created in a word processor or HTML editor, so it was 
necessary to use a “neutral” text format for representing test information. In order to 
make the source file for the test compatible with HTML editors, e.g., Microsoft Front- 
Page, the question/answer portion of the test was specified as an encoding as ordinary 
text in HTML, where < and > are replaced by { and }. This way, an ordinary HTML 
document with an embedded test is still valid HTML and at the same time its test con¬ 
tent would be visible (see example in Appendix B). A subset of the XML DTD is used 
for the embedded text in HTML: <title>, <prologue>, and <epilogue> are 
inferred from the HTML, rather than specified by the test creator. The test format is 
defined in Section A. 
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Figure 8: Page for the generic test, after a response has been submitted 
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Netscape; Sample Ordinary Test 


File Edit View Go Communicator Help 

Bookmarks ^ Go 70:jjhttp://peb. cs. emu. edu/egi-bin/testgen. cgi Related £2 


Bookmarks Go To: fjittp: //peb cs. emu. edu/cgi-bin/testgen. cgi 


' 4 * 3 £ *. a ai m ■ 

Back Reload Home Search Netscape Print Security Shop Slop 


Sample Ordinary Test 

Below is a sample ordinary test 
1 What is the speed limit? 

dISii' vA; 45 


" ^C.6$ ::: ■ 

vD.70 

2. What time is noon? 


vA. iptoo 
vB. nm . 

A C. 12:00 
vD. 1:00 

3, What has three sides? 
:: yA, Triangle A" 

Square 11 
y^a Circle# 


That was the test. 


K :| \«fc- : -S dS3: 



16 

























Figure 10: Page for an ordinary test, after a response has been submitted 
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To upload a new test, the instructor goes to the Web page for uploading tests, and 
specifies a local HTML file to upload (see Figure 11). The software extracts the XML 
portion (encoded as HTML annotations) and parses it with a validating parser, before 
saving out a binary representation of the complete HTML test (which may include 
links, images, etc.), and then indicate that uploading was successful (Figure 12). 

Note: with Prof. Yaron’s class, test submission was not used, because of the sim¬ 
plified nature of the “generic” concept test, which does not have question or answer 
text. 

63 Monitor test 

The “monitor test” user interface consists of the following components: 

• instructions to the student, 

• a bar graph of current student responses, indicating how many responses there 
were for each choice, and 

• buttons and text fields for the instructor to control what is displayed and the score 
database. 

The page for starting the monitoring of an “ordinary test” is in Figure 15. The page 
for starting the monitoring of a “concept test” is in Figure 13. The page showing the 
bar chart used for monitoring the “generic concept test” is in Figure 14); note that this 
page is arrived at directly from a link on the main adminstrative page (Figure 3). 

Instructions to the student were included on this page as a useful reminder of the 
URL that the student needs to load in order to take a test, and a brief summary of how to 
submit responses. The instructions were put in after the first trial because some students 
came in late, or didn’t hear the instructor describe the procedure, or were unfamiliar 
with what was expected. 

When “auto refresh” is turned on, the Web page is updated every 3 seconds, in order 
for the students and instructor to see an updated graph of the latest student responses. 
There is a button to turn on or off auto refresh, because some students said it was 
distracting to see the page refresh at the beginning when either no or few students had 
yet submitted responses. Originally, the default behavior was to have a test begin with 
auto refresh, but this was changed so that the instructor could set up a test to begin 
without having to immediately turn off auto refresh. 

There is a “start/stop” button to allow the instructor to either start a test, or once 
it is in progress, stop it, meaning that when the test is stopped, no more submissions 
are accepted from students, and refresh is turned off. This was to prevent a problem 
that seemed to be occurring in the first trials of the software: some students would 
continue submitting responses after Prof. Yaron had already explained the answer, and 
some students would begin submitting responses for the next question before he had 
reset the test, resulting in those answers being lost. 

For the generic test, there is a text field for setting the number of answers for the 
next test, and also a text field for specifying the name of the score info to be saved out 
upon resetting the score database to begin a new test, an operation performed by the 
“save and reset” button. 
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Figure 12: Page after instructor has successfully uploaded the test 































Figure 13: Page for instructor to specify concept test to monitor 
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Figure 14: Page monitoring generic concept test 
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Figure 15: Page for instructor to specify ordinary test to monitor 























The name of the score data file saved out consists of the current date and time 
followed by the name specified by the instructor (so that if the text field is left blank, 
by default the full name will simply be the date and time). 

The saved data is accessible by means of a separate Web page (Figure 16) that 
pops up a list of score files and prints the scores in a table suitable for post-processing, 
and also recreates, for easy visual summary, the (anonymous) score histogram for the 
particular concept test in question. 

For example, Figure 17 shows a portion of the data for a single test. In the end, most 
students had come up with a final answer of D. The complete history of each student’s 
responses is saved, so we can see from the visible portion of this Web page that some 
students switched to the 4th answer, D, from another answer (and two switched from 
A to D). 


7 Logged results 

The test databases show that 58 students registered their handhelds. 

Figure 18 shows how many students responded to a given number of tests. 

It is encouraging that 14 out of 58 students responded to all the tests that were 
given. The second largest bin occurred at 5 tests out of 11. It would be hard to obtain 
this kind of data about the distribution of student participation without the use of hand¬ 
helds, because the most casual assessment of class participation if human memory and 
hands or flashcards were used, would involve noting for each test how many hands or 
flashcards go up, rather than nothing for each hand whether it goes up for a given test. 


8 Student reactions 

Prof. Yaron at the end of the course gave out a survey for students to fill out, which had 
questions involving the use of handhelds in the course as well as outside it. There were 
50 responses to the survey, a number less than the 58 who registered their handhelds. 

39 students indicated that they reponded to concept tests “frequently (i.e. whenever 
possible)”. The data collected suggests that this is not quite true. Unfortunately, the 
next choice on the questionnaire was “infrequently”, so those who answered half the 
time probably had to choose “frequently”. If the question had provided more bins, e.g., 
“50% of the time”, “75% of the time”, then it might have been interesting to gauge 
students’ own perceptions or memories of their participation against reality. 


9 Prof. Yaron’s Interpretations 

Prof. Yaron’s interpretations of the survey results and assessment of the use of hand¬ 
helds overall: 
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Figure 16: Web page listing saved generic concept test data to display. 
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Figure 17: Example of display of saved concept test data. 
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Number of tests taken 


Figure 18: Number of students (total 58) taking a given number of tests (total 11). 
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me know how confused they are etc. With hands, it is much more difficult to 
know when to start the ”how many people think A... ” process (I have to just 
watch their faces and guess). Cards are easier to manage, so that is probably 
my first choice. But only about 4 students agree with me. 17 students prefer 
show of hands and 28 prefer the handhelds. 

My reading is that the students were neutral to positive on the experience. 

I suspect this would be quite different if they had to buy the devices. 

9.3 Future 

I think the handhelds were fairly useful for giving concept tests. The software 
worked smoothly and students prefer it to other methods. Grading based on 
participation, tried only once this year, could significantly increase student par¬ 
ticipation. However, success of the project probably relies on students being 
given the devices, rather than purchasing them. 

There would be many more uses in the classroom if the handhelds were 
Java enabled. I also can envision many more potential uses for handhelds in 
the laboratories, if students can use them for data collection and analysis. 

10 Implementation 

10.1 Tests 

Each test is, upon being uploaded as annotated HTML, stored both as the original 
HTML and as a binary format through Perl (to avoid having to reparse the original 
source each time). A validation step is currently plugged in, by means of calling an 
external validating parser. There exist XML validators on the Web and in free source 
or binary form; RXP 5 was chosen simply because it is freely available in source form. 
After validation, the Perl library XML:.'Parser (which is not validating, hence the need 
for an external validator) is used to construct a Perl tree representation of the file. 

The Perl library Storable is used to store the Perl object for the parsed test file. 
Each test has a test score database containing both “permanent” and “session” in¬ 
formation. The kind of information stored depends on what kind of test the test is. 
We considered using MySQL for use as a database, but used Berkeley dbm supplied 
with Linux instead, for initial simplicity. In the end, we did not switch to MySQL after 
all, because it turned out that MySQL was known and criticized for its lack of support 
for efficient concurrent read and write access, so it did not seem there would be an 
advantage in using it 6 . 

Each test score database is saved out using Perl’s libraries MLDBM (for dbm), 
DB.File, and Storable. It was later discovered only when moving the software 
to a Solaris SPARCstation, unfortunately, that the resulting database file is not portable 
across platforms, because of byte order issues. The test score database contains the test 
type (concept or ordinary), an ID, to distinguish it from other tests, a flag to indicate 

5 http://www.cogsci.ed.ac.uk/~richard/rxp.html 

6 http://openacs.org/why-not-mysql.html 
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- Storable, 

- CGI (newer version than that which came with the version of Perl installed) 
• rxp XML parser and validator. 

Course directory access permissions also needed to be set up, and the protect Web 
access for the admin page for instructors. 


12 Future work 

One possibility for improving the concept test experience for students is to allow for 
a voting of whether students are confused about a question. Several times Prof. Yaron 
asked the class for a show of hands on who was confused: this information might be 
useful to display and record online, as feedback for the lecture and for the future. 

One idea that was brought up by Prof. Yaron was to provide an “explorer” type of 
user interface for creating quizzes from some large database of questions, i.e., there 
would be a “folder” for each quiz, and questions could be dragged and dropped into 
such folders. This is particularly useful when there is a large body of question material 
and one wishes to easily reuse them to make different tests from them without having 
to physically copy and embed question text into each desired test. And if the database 
of questions kept track of which questions were relevant to which topics, creation of 
tests would be made more convenient. 

Security concerns were hardly addressed during this initial implementation. They 
would be particularly important if online tests became used for significant grading. 

If more and more features are added to the software, it will probably be wise to use a 
“real” database as a repository of all course information, rather than use a combination 
of different file formats and file system hierarchies. 


13 Conclusions 

The students and the instructor in the experimental use of handhelds in the classroom 
appeared to be comfortable with the technology, and the administration of tests and 
submission of responses of tests went smoothly, after the initial phase of adjustment to 
a new process. 

Performance of the Web server based application appeared to be sufficient; at the 
given size of the class and style of use, we did not run into performance bottlenecks 
with the Web server or the wireless network. 

Students appeared to favor the use of handhelds for concept tests to the older meth¬ 
ods of using a show of hands or flashcards. 
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B Sample test 


This is a sample test in HTML form, illustrating the ability to use arbitrary HTML. 

< 1DOCTYPE HTML PUBLIC ” -//IETF//DTD HTML//EN" > 

<html> 

<head> 

<title>Sample Test</title> 

</head> 

<body> 

<hl>Sample Test</hl> 

Below is a sample test: 

<P> 

{test} 

{item} 

{question} 

What is the <em>minimum</em> speed limit? 

{/question}<p> 

{answer correct= n true"} 

45 

{/answer}<p> 

{answer} 

55 

{/answer}<p> 

{answer} 

65 

{/answer}<p> 

{answer} 

70 

{/answer}<p> 

{/item}<p> 

{item} 

{question} 

What <a href="http://www.worldtime.org">time</a> 
is noon? 

{/question}<p> 

{answer} 
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C Chem 106 Concept Test 

The initial 4-question Chem 106 concept test: note empty question and answer text. 

<!DOCTYPE HTML PUBLIC " -//IETF//DTD HTML//EN n > 

<html> 

<head> 

<title>Chem 106 Concept Test</title> 

</head> 

<body> 

{test} 

{item} 

{question} 

{/question} 

{answer} 

{/answer} 

{answer} 

{/answer} 

{answer} 

{/answer} 

{answer} 

{/answer} 

{answer} 

{/answer} 

{/item} 

{/test} 

</body> 

</html> 
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Questionnaire on Handheld Computers 

(Total number of surveys turned in: 50) 

Did you use your handheld computer to: 


Read email: 

never 

infrequently 

frequently 


13 

15 

21 

Browse the web: 

never 

infrequently 

frequently 


5 

22 

22 

Respond to concept tests: 

never 

infrequently 

frequently (i.e. 
whenever possible) 


3 

8 

39 

Maintain contact list (phone # etc.): 

never 

infrequently 

frequently 


42 

1 

1 

Maintain calendar/schedule: 

never 

infrequently 

frequently 


38 

11 

1 

Use Pocket Word: 

never 

infrequently 

frequently 


30 

15 

4 

Use Pocket Excel: 

never 

infrequently 

frequently 


43 

5 

2 

Play games: 

never 

infrequently 

frequently 


1 

23 

20 


Please list any other uses you have found for your handheld: 
- AOL Instant messenger 9 


- mp3 player 2 

- calculator 1 

- notepad 1 

- “to do” list 1 


Please state your level of agreement or disagreement with the following statements: 
I find the concept tests to be a useful part of the lecture 


strongly disagree 

disagree 

agree 

strongly agree 

0 

2 

30 

17 

I usually participate in the concept tests 



strongly disagree 

disagree 

agree 

strongly agree 

1 

2 

22 

24 


I liked using the handhelds for concept tests because the rest of the class could not see my response 
strongly disagree 7 disagree agree strongly agree 

4 13 23 8 

I thought the use of cards was “not cool”, and so prefer either raising hands or using the handhelds 
strongly disagree disagree agree strongly agree 

4 21 17 6 

I thought the use of handhelds for concept tests took too much time 

strongly disagree disagree agree strongly agree 

5 21 19 4 

I thought the presence of handhelds, such that students could use email/browse the web, in the classroom was 
distracting 

strongly disagree disagree agree strongly agree 

4 19 19 7 



Check one of the following: 

19_ I believe that handheld computers will eventually be used widely in courses of this type 

17_ I believe that handheld computers are useful outside of class, but not in class 

8_ I don't see any real use for handheld computers in universities 

Should we use handhelds in the course next year? 27_ yes 17_ no 

What advice would you give to an instructor who wanted to use handhelds in their course? 

- Find more and interesting ways to use them 7 

- Restrict network access during class except to concept tests 3 

- Make sure everyone uses the handhelds 2 

- If the handheld is not used daily, indicate days used 2 

- Money would be better spent on real laptops 2 

- Let us keep the handhelds 2 
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